Initial exploration.
In this notebook we explore data from the Medalla testnet. We are looking at the 388001 first slots.
We use a fork of Lakshman Sankar’s Lighthouse block exporter to export attestations and blocks from the finalised chain until slot 388000.
We present the main datasets below:
all_atsEach row in this dataset corresponds to an aggregate attestation included in a block.
exploded_atsWe cast the dataset above into a long format, such that each row corresponds to an individual attestation included in a block. Note that when this individual attestation is included multiple times over multiple aggregates, it appears multiple times in the dataset.
individual_atsexploded_ats is the “disaggregated” version of the aggregate attestations. To check for validator performance, we often don’t need to check for every inclusion of their individual attestations. individual_ats contains these unique, individual attestations, tagged with some extra data such as their earliest inclusion and whether they attested correctly for the target checkpoint and the head.
Jim McDonald, from Attestant, kindly provided a treasure trove of data on the #medalla-data-challenge channel of the EthStaker Discord server. The two previous datasets could have legitimately been mined from Jim’s data, but we like to get our hands dirty.
all_cmsNot too dirty though: obtaining the past record of committees (which validators are supposed to attend when) is much more computationally intensive, since it requires access to past states. Yet given that we have 12125 epochs in our dataset and a maximum of 73258 validators, a dataset compiling all committee assignments would have 888253250 rows, which is too much. When we need committee information, we’ll pull it from the database and record intermediary datasets instead.
val_balancesThis dataset gives us validator state balances at the beginning of each epoch. Note that the state balance (balance), the true ETH amount a validator deposited, is different from the effective balance (effective_balance), which measures the principal on which validators receive an interest.
To ease the computational demands of this notebook, we record two datasets from which much of the analysis can be derived.
stats_per_valFor each validator, we compute a bunch of statistics, including:
included_ats: The number of times their attestations were includedfirst_att/last_att: The attesting slot of their earliest and latest attestation (used by pintail to build validator types)correct_targets/correct_heads: How many times they correctly attested for the target checkpoint or the headavg_delay: Their average inclusion delaystats_per_slotWe also record summary statistics for each slot. At 388000 slots in our dataset, this remains manageable to query. We have the following fields:
included_ats: How many attestations were received for the slot.expected_ats: How many attestations were expected for the slot.correct_targets/correct_heads: The number of correct target/head attestations for that slot.We compare the number of included attestations with the number of expected attestations.
Clearly something went very wrong circa epoch 2,500. This is now known as the roughtime incident, an issue affecting the major validator client, Prysm. It took time for the network to recover, in the process demonstrating how the quadratic inactivity leak mechanism works. Client diversity FTW!
How many blocks are there in the canonical chain?
Again, the same trough during the roughtime incident.
Attestations vouch for some target checkpoint to justify. We can check whether they vouched for the correct one by comparing their target_block_root with the latest known block root as of the start of the attestation epoch (that’s a mouthful). How many individual attestations correctly attest for the target?
How does the correctness evolve over time?
Attestations must also vote for the correct head of the chain, as returned by the [GHOST fork choice rule]. To check for correctness, one looks at the latest block known as of the attestation slot. Possibly, this block was proposed for the same slot as the attestation att_slot. When the beacon_block_root attribute of the attestation and the latest block root match, the head is correct!
How does the correctness evolve over time?
Validators are rewarded for their performance, and penalised for failing to complete their tasks. We start with a crude measure of performance: the number of included attestations. It is a crude measure since (a) we do not discount the timeliness of the validator, measured by the inclusion delay and (b) we do not check that the attestation’s attributes are correct (with the exception of the source attribute, since an incorrect source cannot possibly be included on-chain).
We compare the percentage of included attestations with the (possibly negative) reward obtained by the validator.
Who are the validators getting a negative return? We plot the same, showing how long a validator has been in service.
Recently activated validators have a much more balanced uptime-reward curve, with the higher performers getting positive returns. Meanwhile, validators who were active since the beginning tend to have smaller returns. This can be due to validator fatigue (validating for a while, then turning off the rig), but a fair number of early validators have high attestation performance yet low return. The roughtime incident is likely to blame here. Let’s focus on these early validators.